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Post Install Script Debugging 


+ Example extend-compute.xml: 
<?xml version="1.0" standalone="no"?> 
<kickstart> 
<post> 
ech "yo" 28>1 > /tmp/fun 


</post> 


</kickstart> 
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Post Install Script Debugging 


+ Produces an XML syntax error 


# rocks list host profile compute-0-1 > /tmp/ks.cfg 
Traceback (most recent call last): 
File "/opt/rocks/bin/rocks", line 264, in ? 
command.runWrapper (name, args[i:]) 
File "/opt/rocks/lib/python2.4/site-packages/rocks/commands/ init .py", line 1868, in runWrapper 
self.run(self. params, self. args) 


parser. feed (line) 
File "/opt/rocks/lib/python2.4/site-packages/ xmlplus/sax/expatreader.py", line 220, in feed 
self. err handler. fatalError (exc) 


File "/opt/rocks/lib/python2.4/site-packages/ xmlplus/sax/handler.py", line 38, in fatalError 
raise exception 


xml.sax. exceptions.SAXParseException: <unknown>:78:11: not well-formed (invalid token) 
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Post Install Script Debugging 


+ One way to debug the error 


# ROCKSDEBUG=y rocks list host profile compute-0-1 > /tmp/ks.cfg 


</kickstart>[parsel] 


[parsel]<kickstart> 
[parsel] 


[parsel]<post> 
[parsel]ech "yo" 2&>1 /tmp/ fun 
Traceback (most recent call last): 
File "/opt/rocks/bin/rocks", line 264, in ? 
command.runWrapper (name, args[i:]) 


File "/opt/rocks/lib/python2.4/site-packages/rocks/commands/ init .py” 
line 1868, in runWrapper 
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Post Install Script Debugging 


+ Can also use xmllint 


<post> 
ech "yo" 28>1 > /tmp/fun 
</post> 


# xmllint extend-compute.xml 
extend-compute.xml:6: parser error :xmlParseEntityRef: no name 
ech "yo" 2&>1 > /tmp/fun 

A 
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Post Install Script Debugging 


+ Fix your syntax error in your XML file 


<post> 
ech "yo” > /tmp/fun 
</post> 


+ But there still is a problem with the post 
script 
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Post Install Script Debugging 


 Aiter a node installs, there are several log 
files saved on the host 
2 À key log file is: /var/log/rocks-install.log 
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Post Install Script Debugging 


+ Execute: 

# rocks list host profile compute-0-0 > /tmp/ks.cfg 
+ Then examine /tmp/ks.cig: 

$Spost --log=/mnt/sysimage/var/log/rocks-install.log 


ech "yo" > /tmp/fun 


+ Output from the commands in the post 
section will be saved to /var/log/rocks- 
install.log on the node's hard disk 
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Post Install Script Debugging 


+ Huh? 
> The filename says /mnt/sysimage/var/log/ 
rocks-install.log 
+ The installer runs inside a ramdisk 


> The ‘root’ (e.g., 7°) for the installer is a 
ramdisk 


> The installer mounts the hard disk partitions 
under /mnt/sysimage 
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Post Install Script Debugging 


+ After the node installs, examine /var/log/ 
rocks-install.log 


./nodes/extend-compute.xml: begin post section 
/tmp/ks-script-MP2uTd: line 2: ech: command not found 
./nodes/extend-compute.xml: end post section 
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Post Install Script Debugging 


+ After the node installs, examine /var/log/ 
rocks-install.log 


./nodes/extend-compute.xml: begin post section 
/tmp/ks-script-MP2uTd: line 2: ech: command not found 
./nodes/extend-compute.xml: end post section 
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Post Install Script Debugging 


+ Put a “stall” in a post section 


<post> 
ech "yo" > /tmp/fun 


while [ -f /tmp/fun ] 


do 

sleep 1 
done 
</post> 
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Post Install Script Debugging 


+ When the node reinstalls, it will stall at the 
“Running post-install scripts” section 


+ Login to the installing node: 


# ssh compute-0-0 -p 2200 


© 2010 UC Regents 13 





© 


# ps auwx 


root 
root 
root 
root 


GO © © GO 
© W © GO 


+ Get list of processes: 


0.0 0 
0.0 65888 
0.0 62616 
0.0 10764 


Post Install Script Debugging 


10:30 
10:30 
12:54 
12:54 
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/bin/sh /tmp/ks-script-THsZwm 
sshd: root@pts/1 
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Post Install Script Debugging 


+ Let's look at the /tmp/ks-script-* file 


# cat /mnt/sysimage/tmp/ks-script-THsZwm 
ech "yo" > /tmp/fun 

while [ -f /tmp/fun [ 

do 


sleep 1 
done 


+ This is the contents of our post script 
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Post Install Script Debugging 


+ We can see what anaconda is doing by 
looking at /tmp/anaconda.log 


# tail -f /tmp/anaconda.log 


10:30:16 INFO : moving (1) to step dopostaction 

10:30:16 INFO : Running kickstart %%post script (s) 

10:30:20 ERROR : Error code 127 encountered running a kickstart 
spre/%Spost script 

10:30:26 ERROR : Error code 9 encountered running a kickstart 


spre/%Spost script 
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Post Install Script Debugging 


Remove the file /mnt/sysimage/tmp/fun 


# rm -f /mnt/sysimage/tmp/fun 


+ And the installation will continue 
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vä Holding An Installation After All 
Post Scripts Have Executed 


+ Add the “rocks-debug” kernel parameter 
to the installing kernel 


+ List all “bootactions” 


# rocks list bootaction 
RAMDISK 


ACTION KERNEL 

ARGS 

install: vmlinuz-5.3-x86 64 itrd.img- 5. 3-x86 i 
ize=150000 lang= devfs=nomount pxe kssendmac selinux=0 no FAR ksdevice=bootif 


ks ramdisk_size= 
install headle vmli -5.3-x86 64 itrd.img-5.3-x86 64 
s ramdisk_ BE A la AE devfs=nomount pxe kssendmac selinux=0 no RN headle 
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vä Holding An Installation After All 
Post Scripts Have Executed 


+ Greate a new bootaction called “install 
debug” based on the “install” bootaction 

# rocks add bootaction action="install debug" N 
kernel="vmlinuz-5.3-x86 64” ramdisk="initrd.img-5.3-x86 64 \ 


args="ks ramdisk_size=150000 lang= devfs=nomount pxe \ 
kssendmac selinux=0 noipv6 rocks-debug" 
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vä Holding An Installation After All 
Post Scripts Have Executed 


+ Assign “install debug” installaction to a 


node 
t rocks list host 
HOST MEMBERSHIP CPUS RACK RANK RUNACTION INSTALLACTION 
build-x86-64: Frontend 8 0 0 os install 
compute-0-1: Compute 8 0 1 os install 
compute-0-0: Compute 8 0 0 os install 


# rocks set host installaction compute-0-0 action="install debug” 


# rocks list host 


HOST MEMBERSHIP CPUS RACK RANK RUNACTION INSTALLACTION 
build-x86-64: Frontend 8 0 0 os install 
compute-0-1: Compute 8 0 1 os install 
compute-0-0: Compute 8 0 0 os install debug 
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vä Holding An Installation After All 
Post Scripts Have Executed 


+ The PXE configuration file is updated with 
the “rocks-debug” flag 


# cat /tftpboot/pxelinux/pxelinux.cfg/0A01FFFE 

default rocks 

prompt 0 

label rocks 
kernel vmlinuz-5.3-x86 64 
append ks ramdisk_size=150000 lang= devfs=nomount pxe 
kssendmac selinux=0 noipv6 rocks-debug 
initrd=initrd.img-5.3-x86 64 
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Post Install Script Debugging 


@ When the node reinstalls, it will stall at the 
“initial X screen” 
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Post Install Script Debugging 


+ At this point, all post scripts have 
executed and the installation is stalled 


2 Login to the node and you'll see: 


# ssh compute-0-0 -p 2200 


-bash-3.2# tail -f /tmp/anaconda.log 


10:54:01 ERROR : Error code 127 encountered running a kickstart %pre/%post script 
10:54:07 ERROR : Error code 9 encountered running a kickstart %pre/%post script 
10:54:08 INFO : All kickstart %%post script(s) have been run 
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Post Install Script Debugging 


+ You now have unlimited time to poke 
around on the node 


+ To complete the installation, you need to 
reboot the node (e.g., <Ctrl><Alt><Del>) 
> Or, if you logged in via “ssh compute-0-0 —p 
2200” execute: 
# reboot -f 
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© Other Log Files on an /nstalling 
Node 


+ Look at other log files on an installing node 


# ls /tmp/*log 
/tmp/anaconda-http.log /tmp/httpd.log /tmp/syslog 
/tmp/anaconda.log /tmp/ks-pre.log 


+ /tmp/httpd.log 
> Log messages from lighttpd 
+ /tmp/syslog 
< Typical syslog 
+ /tmp/anaconda-httpd.log 
< RPM download timing information 
+ /tmp/ks-pre.log 


> Output from “pre” scripts 
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!tmp/httpd.log 


127.0.0.1 127.0.0.1 - [28/May/2010:10:45:11 -0700] 

"GET /install/rocks-dist/x86 64/RedHat/RPMS/dmraid- 
events-1.0.0.rc13-53.e15.x86 64.rpm HTTP/1.1" 200 18083 
"-" "urlgrabber/3.1.0 yum/3.2.22” 


127.0.0.1 127.0.0.1 - [28/May/2010:10:45:13 -0700] 
"GET /install/rocks-dist/x86 64/RedHat/RPMS/compat- 
libf2c-34-3.4.6-4.1386.rpm HTTP/1.1" 200 2674 "-" 
"urlgrabber/3.1.0 yum/3.2.22" 
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/tmp/syslog 


<4>raid6: 
<4>raid6: 
<4>raid6: 
<4>raid6: 
<4>raid6: 
<4>raid6: 
<6>md: raid6 personality registered for level 6 
<6>md: raid5 personality registered for level 5 
<6>md: raid4 personality registered for level 4 


int64x4 1929 MB/s 
int64x8 1585 MB/s 
sse2x1 3195 MB/s 
sse2x2 5332 MB/s 
sse2x4 6058 MB/s 
using algorithm sse2x4 (6058 MB/s) 
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/tmp/anaconda-httpd.log 


1275068709.09: RedHat/RPMS/dmraid-events-1.0.0.rc13-53.e15.x86 64.rpm start 
1275068711.1: RedHat/RPMS/dmraid-events-1.0.0.rc13-53.e15.x86 64.rpm end 
1275068711.1: RedHat/RPMS/compat-libf2c-34-3.4.6-4.1386.rpm start 
1275068713.1: RedHat/RPMS/compat-libf2c-34-3.4.6-4.1386.rpm end 
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/tmp/ks-pre.log 


/tmp/ks-script-mvCzD5: line 12: /opt/rocks/bin/ci: No such file or directory 
/tmp/ks-script-mvCzD5: line 13: /opt/rocks/bin/co: No such file or directory 
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Log Files on an /nstalled Node 


+ After a node is installed, several log files are saved under “/root” 


# ls /root/*log 
/root/httpd.log /root/install.log.syslog /root/rocks-pre.log 
/root/install.log /root/rocks-post.log 


+ /root/httpd.log 
> A copy of the log messages from lighttpd 
+ /root/install.log 
> A list of RPMs installed by anaconda 
+ /root/install.log.syslog 
> Post script error messages 
+ /root/rocks-pre.log, /root/rocks-post.log 
> Output from rocks-pre.sh and rocks.post.sh scripts 
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/root/install.log 


Installing hal-0.5.8.1-52.e15.1386 
Installing kudzu-1.2.57.1.21-1.e15.centos.x86 64 
Installing hal-0.5.8.1-52.el5.x86 64 


Installing system-config-network-tui-1.3.99.18-1.e15.noarch 
Installing firstboot-tui-1.4.27.7-1.e15.centos.x86 64 


+ Shows the packages that were installed 


> And it shows the “architecture” of the 
package 
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/root/install.log.syslog 


<86>May 28 10:53:53 useradd[3907]: new group: name=nfsnobody, GID=4294967294 
<86>May 28 10:53:53 useradd[3907]: new user: name=nfsnobody, UID=4294967294, 
GID=4294967294, home=/var/lib/nfs, shell=/sbin/nologin 

<86>May 28 10:53:53 useradd[3925]: new group: name=haldaemon, GID=68 

<86>May 28 10:53:53 useradd[3925]: new user: name=haldaemon, UID=68, GID=68, 
home=/, shell=/sbin/nologin 


+ | never look at this file! 
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/root/rocks-pre.log /root/rocks-post.log 


+ On the first boot after an installation, the following scripts are run: 


/root/rocks-pre.sh 
/root/rocks-post.sh 


The scripts ensure the file permissions set with “<file>” tags are 
maintained 


Node XML file: 


<file name="/etc/rc.d/init.d/mdmonitor" perms="755”> 


/root/rocks-pre.sh: 


if [ -f /etc/rc.d/init.d/mdmonitor ]; then 
echo "rocks" | /opt/rocks/bin/ci /etc/rc.d/init.d/mdmonitor; 
/opt/rocks/bin/co -f -1 /etc/rc.d/init.d/mdmonitor; 

fi 

chmod 755 /etc/rc.d/init.d/mdmonitor 
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/root/rocks-pre.log /root/rocks-post.log 


+ The log files mainly contain RCS status 


messages 
> All files modified with a “<file>” tag are managed with 
RCS 
/etc/rc.d/init.d/RCS/mdmonitor,v <-- /etc/rc.d/init.d/mdmonitor 
new revision: 1.2; previous revision: 1.1 
done 


/etc/rc.d/init.d/RCS/mdmonitor,v --> /etc/rc.d/init.d/mdmonitor 
revision 1.2 (locked) 
done 
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“Lights Out” Frontend 
Installation 
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(E || EBSC 
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“Lights Out” Frontend Install 


+ Fu | ly-auto M ated build-x86-64.rocksclusters.org 
frontend install 


C 
Install a virtual frontend ein 
(vi-1) inside a physical 
frontend (build-x86-64) 
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“Lights Out” Frontend Install 


+ How it works: 


> On anormal installation: 


٠ After the user selects the rolls, a file named /tmp/rolls.xml 
is created 


٠ After the user inputs all data in the user “screens” (e.g., 
“Enter Private Network Configuration”), a file named /tmp/ 
site.attrs is created 


> So, if the files /tmp/site.attrs and /tmp/rolls.xml 
already exist on the ramdisk, then the machine will 
bypass the user screens and automatically install 
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Lights Out Frontend Installation 


+ Create a virtual frontend 


rocks add cluster vi-1.rocksclusters.org 137.110.119.118 0 


+ Set attributes 


rocks set host attr vi-1.rocksclusters.org \ 
Kickstart PrivateHostname vi-1 


rocks set host attr vi-l.rocksclusters.org ١ 
Kickstart PublicAddress 137.110.119.118 


rocks set host attr vi-l.rocksclusters.org \ 
Kickstart PublicHostname vi-l.rocksclusters.org 


rocks set host attr vi-l.rocksclusters.org \ 
Kickstart PublicKickstartHost build-x86-64.rocksclusters.org 
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Assign Rolls to the New 
Frontend 


# rocks set host roll vi-l.rocksclusters.org os 5.3 x86 64 
# rocks set host roll vi-l.rocksclusters.org base 5.3 x86 64 
# rocks set host roll vi-l.rocksclusters.org kernel 5.3 x86 64 


# rocks list host roll vi-1.rocksclusters.org 


HOST NAME VERSION ARCH OS 

frontend-0-0-0: os 5 13 x86 64 linux 
frontend-0-0-0: base 5 13 x86 64 linux 
frontend-0-0-0: kernel 5.3 x86 64 linux 
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/tmp/rolls.xml 


# rocks report host roll vi-1.rocksclusters.org 


<rolls> 
<roll 
name="os" 
version="5.3" 
arch="x86 64" 
url="http://build-x86-64 .rocksclusters.org/install/rolls/" 
diskid="" 
/> 
<roll 
name="base" 
version="5.3" 
arch="x86 64" 
url="http: //build-x86-64.rocksclusters.org/install/rolls/" 
diskid="" 
/> 
<roll 
name="kernel" 
version="5.3" 
arch="x86 64" 
url="http://build-x86-64 .rocksclusters.org/install/rolls/" 
diskid="" 
/> 
</rolls> 
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/tmp/site.attrs 


# rocks report host attr vi-1.rocksclusters.org 


Kickstart PublicAddress:137.110.119.118 

Kickstart PublicBroadcast:137.110.119.255 

Kickstart PublicDNSDomain:rocksclusters.org 

Kickstart PublicDNSServers:8.8.8.8 

Kickstart PublicGateway:137.110.119.1 

Kickstart PublicHostname:vi-l.rocksclusters.org 

Kickstart PublicKickstartHost:build-x86-64.rocksclusters.org 
Kickstart PublicNTPHost:pool.ntp.org 
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kud Need to Create rolls.xml and 
Site. attrs 


+ Create node XML file ‘extend-wan.xml’ 


>In the directory: 
* /export/rocks/install/site-profiles/5.3/nodes 
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kud Need to Create rolls.xml and 
site.attrs Files in the Installer 


<?xml version="1.0" standalone="no"?> 

<kickstart> 

<pre> 

<file name="/tmp/site.attrs"> 

<eval>/opt/rocks/bin/rocks report host attr &hostname;</eval> 
</file> 

<file name="/tmp/rolls.xml"> 

<eval>/opt/rocks/bin/rocks report host roll &hostname;</eval> 
</file> 


</pre> 


</kickstart> 
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kud Rebuild the Distro and Boot the 
VM 


# cd /export/rocks/install 
# rocks create distro 


t rocks start host vm vi-l.rocksclusters.org 


+ 10 minutes later, you've got a new 
frontend! 
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Fin 
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Partitioning 
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Partitioning 


+ You can see what the “programmatic 
partitioning” will do 

+ Login to compute node and execute: 
# ssh compute-0-0 p 2200 
-bash-3.2# /tmp/product/do partition.py 
part / --fstype ext3 --onpart sdal 
part /var --fstype ext3 --onpart sda2 


part swap --noformat --onpart sda3 
part /state/partitionl --noformat --onpart sda5 
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Partitioning 


+ If you have a bug in your programmatic 
partitioning code, you'll need to recreate the 
code from the ‘pre’ script 

> On the frontend, create /tmp/ks.cfg 


# rocks list host profile compute-0-0 > /tmp/ks.cfg 
> Find the ‘pre’ script in /tmp/ks.cfg with your 
partitioning code 


+ Make a new file named /tmp/a.py from the 
partitioning code 


© 2010 UC Regents 48 





/tmp/a.py 


#! /opt/rocks/bin/python 
import rocks partition 


membership = 'Compute' 
nodename = 'compute-0-1' 


def doDisk(file, disk): 
file.write('clearpart --all --initlabel --drives=%s\n' % disk) 
file.write('part / --size=6000 --fstype=ext3 --ondisk=%s\n' % disk) 
file.write('part /var --size=2000 --fstype=ext3 --ondisk=%s\n' % disk) 
file.write('part swap --size=2000 --ondisk=%s\n' % disk) 
file.write('part /mydata --size=1 --grow --fstype=ext3 --ondisk=%s\n' 
% disk) 


# 


# main 


# 


p = rocks partition.RocksPartition () 
disks = p.getDisks () 


if len(disks) == 
file = open('/tmp/user partition info', 'w') 
doDisk (file, disks[0]) 
file.close () 
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© Copy the Script to the Compute 
Node 


+ Put the script in /tmp/product on the 
compute node 


-bash-3.2# scp 10.1.1.1:/tmp/a.py /tmp/product/a.py 


+ Run it: 


-bash-3.2# ./a.py 


-bash-3.2# cat /tmp/user partition info 

clearpart --all --initlabel --drives=sda 

part / --size=6000 --fstype=ext3 --ondisk=sda 

part /var --size=2000 --fstype=ext3 --ondisk=sda 

part swap --size=2000 --ondisk=sda 

part /mydata --size=1 --grow --fstype=ext3 --ondisk=sda 
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